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The inventioii relates to a method of converting of a first set of initial 

segments of an image into a second set of updated segments of the image, the method 

comprising iterative updates of intermediate segments being derived from respective initial 

segments, a particular update comprising determining whether a particular pixel being 

located at a border between a first one of the intermediate segments and a second one of tiie 

intermediate segments, should be mov^ from die first one of the intecmediale segments to 

the second one of tiie intermediate segments, on basis of a pixel value of tiie particular pael, 

on basis of a first parameter of the first one of flie intermediate segments and on basis of a 

second parameter of the second one of the intermediate segments. 

The invention fiirther relates to a conversion unit arranged to perfoim such a 
method of converting. 

The invention fiirflier relates to an image processing ^aratus, comprising: 

- receiving means &x receiving a signal representing an image; 

- a segmentation unit for determining a first set of initial segments of the 

image; 

- a conversion unit for converting the first set of initial segments into a second 
set of updated segments; and 

- an image processing unit for processing die image on basis of the second set 
of updated segments. 



Image segmentation is an important first step that often precedes oflier tasks 
such as segment based d^ estimation or video compression. Generally, image 
segmentation is the process of partitioning an hnage into a set of non-overlapping parts, or 
segments, tiiat together correspond as much as possible to die physical objects that are 
present in tiie scene. There are various ways of ^proaching the task of image segmentation, 

including histogram-based segmentation, edge-based segmentation, region-based 
segmentation, and hybrid segmentation. 
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The method of fhe kind described in fhe opening paragraph is known in fhe art. 
Wiih Ifais known method a first set of initial segments of an image is converted into a second 
set of iqidated segments' of the image. The mefliod comprises iterative updates of 
intermediate segments being derived ftom respective initial segments. An ijpdate comprises 
5 determining whether a particular pixel being located at a border between a first intermediate 
segment and a second intermediate segment should be moved from the first intermediate 
segment to the second intermediate segment This is based on the color value of the particular 
pixel, the mean color value of the first intermediate segment and on basis of the mean color 
value of the second intermediate segment. If it appears that the particular pixel should be 
10 moved from the first intermediate segment to the second intermediate segment new mean 
color values are computed for the new intermediate segments. Siibsequently a next pixel is 
evaluated and optionally moved. After evaluation of the relevant pixels of the image in one 
scan over the image, another scan of evaluations over the image is started. 

The known method however suffers from the &ct that several segmentation 
1 S refinement iteraticms of the complete image have to be performed for realizing pixel-precise 
segmentation. Typically, twenty scans over the image are made to achieve the second set of 
updated segments of the image. This approach is therefore very e>q)ensive in terms of 
memory access, power consumption and computational effort. 



20 

It is an object of the invention to jnrovide a method of the kind described in the 
opening paragraph which is relatively efBcient with regard to memory access. 

This object of the invention is achieved in that first a number of iterative 
tq^dates are performed for pixels of a Grst two-dimensional block of pixels of the image and 

25 after tliat the number of iterative updates are performed for pixels of a second two- 
dimensional block of pixels of the inoiage. Typically the dimensions of the blocks of pixels are 
S'^'S or 16*^16 pixels. The evaluations are performed for the relevant pixels in a block in a 
number of scans. That means that, e.g. row by row these relevant pixels in the block under 
consideration a evaluated and after that again relevant pixels of that block are evaluated. Note 

30 that the parameters of the segments are adapted after each evaluation. After the relevant 
pixels of a block of pixels have been evaluated in a number of scans, the pixel values of 
another block of pixels are evaluated in a similar way. With relevant pixels is meant those 
pixels which are located at a border between two segments. Note that a border moves, i.e. the 
edge of a segment changes, if a pixel is taken firom an intermediate segment and added to its 
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neig^orixig infennediate segmenL Thetefiur Ibe relevant pixels of a block is diOerent for 
each of the scans. 

An advantage of the niethod accofding to the inventiiDh 
window, conqnising the pixels of subsequent blocks, is moved over the image only once. 
5 That means tiiat Ihe blocks of pixels have to be accessed only once froma memory device 
Typically the pbtel vahies of a block under consideration are tenq)oraiily stored in a cache. 
Then the iterations are performed on basis of the values in the cache. 

In an embodiment of the mefliod according to flie invention, the first parameter 
corresponds to a mean color value of the first intermediate segment^ the second parameter 
10 corresponds to a mean color value of flie second intermediate segment and the pixel value of 
the particular pixel r^iresents the color value of the particular pfaceL Color is a relatively 
good criterion fiw unage segmentation. An advantage of this embodiment accordmg to the 
invention is that flie updated segments relatively well correspond to objects in the scene. 

In an enibodiment of the method according to the invention, the particular 
15 i^f^isbasedonaregularizationteimdependmgonthesht^offliefirstQneofthe 

intermediate segments, the regularization term being computed on basis of a first group of 
pbcels of the first two-dimensional block of pfacels. In other words, the regularization term 
depends on the sh^e of the boundary between segments. The regularization term penalizes 
uregular segment boundaries. An advantage of this embodiment according to the invention is 
20 that relatively regular segment boundaries are determined. Therefor this embodiment 
according to the invention is less sensitive to noise in the image. 

In an embodiment of the method according to iSao invention, a first sequence of 
the number of iterative updates are performed in a row-fay-row scanning within the first block 

of pixels and a second sequence of the number of iterative iqpdates are performed in a 
25 column-by-column scanning within the first block of pbcels. lii other woids, the scanning 
directions are alternated between successive scans. For instance, first a scan in a horizontal 
dhsction is perfiMmed and then m vertical direction. Alternatively, first a scan in a vertical 
dhsction is perfimned and then in horizontal duection. Optionally, a third scan is in the 
opposite direction of tiie first scan, e.g. left-to-right versus right-to-left. Optionally, a fourth 
30 scan is in the opposite direction ofthe second scan, e.g. top-to-bottom versus bottomt-to-top. 
Preferably the values of the regularization terms are different for tiie various scans, e.g. 
starting from a low curvature penalty to a high curvature penalty. 

In an embodiment of the method accordmg to the invention tiie first two- 
dimensional block of pixels is located adjacent to tiie second two-dhnensional block of 
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pixels. An advantage of this embodiment according to the invention is that a relatively simple 
memory allocation scheme is achieved. 

In an embodiment of the method according to the invention the regularization 
term is computed on basis of the first group of pixels of the first two-dimensional block of 
pixels and a second group of pixels of the second two-dimensional block of pixels. By also 
takmg mto account pixels of a neigjiboring block of pixels a better regularization term can be 
computed for pixels at flie border of a block. 

It is a further object of the invention to provide a conversion unit of the kind 
described in the opening paragraph which is relatively efficient with regard to memoiy 
acc^. 

This object of the invention is achieved in that the conversion unit conges 
computation means far perfotming first a number of iterative t^dates for pfacels of a first two- 
dunensional block of pixels of the image and fi»r, after fliat, performing die number of 
iterative updates for pixels of a second two-dhnensional block of pixels of the image. 

It is advantageous to apply an embodiment of the conversion unit according to 
the invention in an image processmg apparatus as described in the opening paragraph. The 
image processing apparatus may comprise additional components, e.g. a display device for 
displaying the processed images or storage means for storage of the processed images. The 
image processing unit might support one or more of the following lypes of image processing: 

- Video conapression, Le. encoding, e.g. according to the MPEG standard or 
H26L standard; or 

- Conversion of traditional monoscopic video (2D) video material into 3D 
video for viewing on a stereoscopic (3D) television. In this technology, stracture ftom motion 
methods can be used to derive a depth m^ fiom two consecutive hnages in the video 
sequence; or 

- Image analysis for e.g. vision-based control like robotics or security 

applicati<uis. 

Modifications of the method and variations thereof may correspond to 
modifications and variations thereof of the conversion unit and of tiie image processing 
apparatus described. 



These and other aspects of the metiiod, of the conversion unit and of the image 
processing ^aratus according to the mvention wiU become apparent fiom and will be 



29.04.2003 

elucidated with respect to the implementations and embodiments described hereinafter and = 
with reference to the aocQnq)anying drawings,, wherein: 

Kg. 1 sdiematically shows the scanning scheme according-to the prior ar^ 
Kg. 2 schematicaUy shows the scanning scheme according to the invention; 
K&3schematicaDyshowsflieupdateoftwoneighboringintermediate. ' 



segments; 
pixels; 



Fig. 4 schematically shows subsequent scaoning directions fi>r a block 



of 



Fig. 5 schematicaUy shows a sliding window of a number of blocks; 

Fig. 6 schematicaUy shows an image processing ^>paratus accordh^ to the 

invention; and 

Fig. 7 schematicaUy showsanumber of components in the context of a 
conversion unit according to the invention. 

Same refer^ice numerals axe used to denote i 



An important step m converting 2D video to 3D video is the identification of 
m«ge segments or regions with homogeneous color, Le., image segmentation. Depth 
discontinuities are assmned to coincide with the detected edges of homogeneous color 

regions. A single dqrth value is estimated for each color region. This dq>th estim^ 
region has flie advantage timt there exists per definition a large color contrast along the region 
bomidaiy. The temporal stabiUty of color edge positions is critical for ti» final quality of the 
depfli maps. When die edges are not stable over time, an amioying flicker may be perceived 
by flie viewer when tiie video is shown on a 3D cok>r television. Thus, a time-stable 
segmentation metiiod is tixe first step m ti« conversion process ftom2D to 3D video. Image 
segmentation using a constant color model achieves tiiis desired e^ This me&od of image 
segmentation is described in greater detafl below. It is based on a first set of initial segments 
and Iterative iq,dates resulting in a second set of updated segments. In oAer words the 
segmentation is a conversion of a first set of initial segments mto a second set of updated 
s^ments. 

The constant color model assmnes that flie time-varying image of an object 
segment can be described in sufficient detail by tiie mean region color. An image is 
represented by a vector-vahied fiinction of image coordinates: 
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where r(x,y) , gix^y) b(x^y) are the red, green and blue color channel. The object is to 
find a region partition referred to as segmentation L consisting of a fixed number of 
segments N . The optimal segmentation L^^^ is defined as the segmentation that minimizes 

Ihe sum of an error term 6(j[:,>;)plus aregularizationterm fix^y) over all pixels in the 
S image: 

L,^ =argnmiX;<x,y)+;fcX;/(x,j;) (2) 

where A: is a regularization parameter that weights the imqportance of the regularization terra 
The regularization term limits the influence that random signal fluctuations, such as sensor 
noise, have on the edge positions. The error e{x, y) at pixel position (x^y) depends on the 

1 0 color value I(jc, y) and on the segment label L(x, y) : 

<^,J')H|I(^,J')-mz(,,,)ll2 (3) 
where m^^^^ ^y is Hie mean color for Hie segment with label L(x,y) . The subscript at the 
double vertical bars denotes the Euclidean norm. The regularization term f(x^y) depends on 
the shape of the boundary between segments: 
15 nx,y) = '£^iLix,y)Mx\y) (4) 

where (x',y ) are coordinates firom the 8-connected neighbor pixels of (x^y) . The value of 
x(A,B) depends on whefiier segment labels A and B dififen 

0 otherwise 

Function f(x,y) has a straightforward interpretation. For a given pixel 

20 position (x^y) , the function simply returns the numb» of 8-connected neighbor pixels that 

have a different segment label. 

Given the initial segmentation, a change is made at a segment boundary by 
assigning a boundary pixel to an adjoining segment Suppose that a pixel with coordinates 
(x,y) cuirentiy in segment with label A is tentatively moved to segment witii label B . Then 

25 the change in mean color for segment A is: 

_ my)-m, (6) 

and the chan^ in mean color for segment B is: 
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(7) 

where and are the number of pixels inside segmente ^ and 5 respectively. The 
pwqwsedldjel change causes a coireqjonding change in the error fim^^^ 

5 The proposed label change fiom ^ to i? atpixel (x,y) also changes the global regulaiization 
fimction /. The proposed move affects / notonlyat (x,:K),butalsoatflie 8-connecled 
neig^ibor p«el positions of ix,y). The change in regularization fimction is given by the sum 

= *AE/(x,3;) = 2 £ )) - x(^Z(x',y))] (9) 

\*^} \x»y} ixyy 

where (x'.y) are the 8-connected nei^U)or pixels of (x,:j;) . 
^ "^^^ Proposed label change improves the jSt criterion if 

(^) (t:?) ^'"^ 

Fig. 1 schematically shows the scanning scheme according to the prior art. Fig. 
1 shows image wife intermediate segments A.B,C and D being derived ftom initial segments 
fiom the begmnmg of the conversion and the same image with the updated segments 

A'3%C» and D\ The pixels of image are evaluated in a line-by-Bne scanning as indicated 
with the arrows, e.g. arrow 102. Afler one scan over the image a subsequent scan over tiie V 
hnage is performed. The evaluation is based on the evaluation of color models as described 
above. 

Fig, 2 schematically shows the scanning scheme according to the invention. 
Fig. 2 shows an image with intermediate segments A,B,C and D behis derived fiom mitial 
segments fi»m the bcginnmg of the conversion and the same image with the updated 
segments A»3%C' and D'. The pixels of the image are evaluated m a block by block scheme. 
That means tiiat first a number of iterative evaluations are performed for the relevant pixels 
wiflrin a first block 200. After that a nmnber of iterative evataations are performed for the 
relevant pfacels witinn a second block 202. The direction of the scamnng witiiin a block might 
be as depicted with arrow 204, i.e. row-by-^ow. The evaluations are based on tiie evaluation 
of color models as described above. 

Fig. 3 schematically shows tire update of two neighboring \rMpr«^^^ 
segments AandB into A' andB% respectively. Fig. 3 schematicaUy shows a block 200a of 
8*8 pixels which is located at a border 302 between a first mtermediat« segment A and a 
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second intexmediate segment B. The pixel 300 wiHi coordinates (x, y) is evaluated. That 
means it is detemiined whether pixel 300 should be moved to segment B. The evaluation is 
based on the computations as specified in Equations 6-9. On basis of the evaluation the pixel 
300 is moved Fig. 3 also shows Ihe same block 200b of 8*8 pixels being located at a border 
5 304 between a third intermediate segment A' and a fomlh intermediate segment B*. The ttiird 
intermediate segment is derived fix>m the first intermediate segment A and the fourth 
intermediate segment B' is derived from the second intermediate segmrat B. 

Fig. 4 schematically shows subsequent scanning directions for a block of 
pixels. The scanning over the pixels in a block for the evaluation might be altematingly in a 
10 horizontal 200a, 200c and vertical direction 200b, 200d. Brides that the scanning can be 
from left-to-right 200a and vice versa 200c. Besides that the scanning can be from top-to- 
bottom 200d and fix>m bottomnto-top 200b. Besides that, a not depicted, zigzag scan is 
possible. 

Fig. S schematically shows a sliding window SOO of a number of blocks 200- 
15 216. Typically these blocks 200-216 are simultaneously cached when the pixels of the central 

block 208 are evaluated The neighboring blocks 200-206 and 210-216 are required for the 

computation of the regularization term as specified in Equation 4. After all evaluations have 

been performed for the central block 208 a new window 502 is defined within the image. 

This new window comprises the blocks 206-222. The central block 214 of this window will 
20 be evaluated now. It should be noted that if there is no edge within a block that then that 

block will be skipped and the window is moved finlher. Within a block only those pixels 

which are located at the border of a segment are evaluated. 

Fig. 6 schematically shows an image processing apparatus 600 according to 

the invention, comprising: 
25 - receiving means 602 for receiving a signal representing video images; 

- a segmentation unit 604 for determining a first set of initial segments of one 
of the video images; 

- a conversion unit 606 for converting the first set of initial segments into a 
second set of updated segments A%B',C*^'; and 

30 - an image processing unit 608 for processing the video image 1 10b on basis 

of the second set of updated segments A'3%C'JD'. 

The input signal may be a broadcast signal received via an antenna or cable but may also be a 
signal from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk 



15 
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(DVD). The mput signal is provided at fte ii^ connector 610. The image processing 
^Jparatus 600 provides Ae ou^ut at the output connector 612. 

The conversion unit 604 for converting the first set of initial segments into a 

secondsetof«pdatedsegmentsm^bein5>Iementedush,goneprocessor.NbnnaU^ 
. ««^<^<»i«P«toedundercontrolofasoWprogramproductDu^^ 

norn^Uy the softv^programproduct is loaded intoamemory.likeaR^ 
from there. Theprogram may be loaded from a background memory, like a ROM, hard disk, 
or magneticany and/or optical storage, or may be loaded via a network like Interna 
Optionally an appKcation specific integrated circuit provides the disclosed functionaKty 

The segmenlation unit 604, the conversion unit 606 and the image processing 
umt 608 can be combined into one processor. 

The output mightbe a stream of con^aessed video data. Alternatively the 

output represents 3Dvideo content The conversion of the received video images mtoft^ 
video content might be as disclosedby M. Qp deBeeckand A. Redert, in "Three 
dm^ensional video for the home", m Proceedmgs of the International Conference on 
Augmented Vhtual Enviromnents and Three-Dimensional Imaging. Myconos. Greece. 2001 
pp 188-191. 

The image processing ^aratus 600 might e.g. be a TV. The hnage processing 
^aratus600mightcompriseadisplaydevice.Alternatively^^ 
600 does not comprise fte optional display device butprovides theou^ut data to an 
^9>paratus that does comprise a display device. Then the image processmg apparatus 600 

nughtbee.g,asettopbox,asatelKte-tuner,aVCRplayer.aDVDplayerorrecorder.TTie 

nnageprocessingapparatus 600 might also beasystembemg^liedbyafihn-studio or 
broadcaster. 

Opa<maaytheimageimKeMi.Bi,pp,«„,«00coG^ 

Hg. 7 sd«natioany showsanuniber «f compomnte 702, 704 m the context of 
aco,™,,i„„„rit706acconJio8.„,l„i„ve«ion.Tl.esya«^ 
de™, for «„«gB of toage daa. e.g. ae lumi^nce «ri color values of the pixels of the 

co«prisesaco»ve«i«.™rf.706which.isa™ngedtooonve«aflm 

an tmage teto a second set of updated segments A-3%C-,D-. This conversion is done by 

»e«»rfite«iveupdatesofin,eo«diatesegn«ntsA3.QDbeingderivedfiom . 
nnt»I segments, whereby aparticular update comprises determining ,Ae,her aparBcuIar 
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pixel 300 being located at a border 302 between a first one of the intennediate segments A, 
and a second one of the intennediate segments B, should be moved firom tibie first one of tiie 
intermediate segments A to the second one of the intennediate segments B, on basis of a 
color value of the particular inxel, on basis of the mean color value of flie first one of the 
5 intermediate segments A and on basis of the mean color value of the second one of die 
intemiediate segments B. The first set of initial segments of an hnage are provided at the 
second input connector 712 and the second set of updated segments A'3%C' are at the 
output connector 714. 

The conversion unit 706 comprises computation means for performing first a 
1 0 nuniber of iterative updates for pixels of a first two-dimensional block of pixels 208 of the 
image and for, after that, poforming the number of iterative iqjdates for pixels of a second 
two-dimensicmal block of pixels 214 of the image. The pixels of the blocks 200-216 are 
simultaneously cached within the cache 704 when the pixels of the central block 208 are 
evaluated. After all evaluations have been performed for the central block 208 a new window 
15 502 is defined within the image. This new window conq^rises the blocks 206-222, The central 
block 214 of Ihis window will be evaluated now. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention and that those skilled in the art will be able to design alternative 
embodiments without departing from the scope of the appended claims. In the claims, any 
20 reference signs placed between parentheses shall not be constracted as limiting the claim. 
The word ^comprising' does not delude the presence of elements or steps not listed in a 
claim. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
coniprising several distinct elements and by means of a suitable programmed computer. In 
25 the unit claims enumerating several means, several of these means can be embodied by one 
and the same item of hardware. 
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CLAIMS: . . 



1. A melhod of converting of a first set (100a) of initial segments of an _ 

into a second set (100b) of ,q,dated segments (A'3'.C'4>') of fl»e image, the method 
comprising iterative updates of intermediate segments (A3.CJ>)bein^ . 
respective initial segments, a particular update comprising determining Aether a particular 
pncel(300)beinglocatedataborder(302)betweena£«stoneof^^ 

(AX and a second one of the intermediate segments (BX should be moved fiom the first one 
of the mtermediate segments (A) to the second one of the intemxediate segments 
of apixel value of Ae particular pixel, onbasis of a first parameter of the first one of the 
ml^miediate segments (A) and on basis of a second parameter of the second one of the 
mtemiediate segments (B), characterized in that first a number of iterative i^dates are 
perfomied for pfacels of a first two-dimensional block of pfacels (200) of the image and afler ' 

thatfl»enumberofiterativeupdatesareperfom.edforpixelsofasecondtwos^ 
block of pixels (204) of the image. 

2. ^"^^«d<>f«»^vertingasclaimedinclaiml,chamcterizedinth^ ; 

parametercortespondstoameancolorvahieafftefirstmtermediatesegmentithesec^^ ^ 

parameter corresponds to a mean color value of flie second intennediate segment and ^ 
P«el value of the particular pixel represents the color value of the particular pixel. 

3. ^«>^°dofconv«tiiigasclaimedinclaimlor2,characterizedintothe 
parbcular^pdateisbasedonaregularizationtermdependingon&^ 

the mtermediate segments, fteregularizationtem. being confuted onbasis ofafi^ 
pixels of the first two-dimensionai block of pixels. 



4. A method of converting as claimed in claim 1 , characterized in Aat a first 

sequence of the number of iterative updates areperformedinarow-by-row^^ 

the first block of pixels andasecond sequence of the nmnber of iterative updates^ 
performed in a coWby-column scamring within the first block of p«els. 



PHNL030441 



12 29.04.2003 
S. A mefhod of cxmverting as claimed in claim 1, characterized in fhat the first 

two-dimensional block of pixels is located adjacent to the second two-dimeiudonal block of 
pixels. 

5. Amethodof converting as claimed in Claim 1, characterized in that the 

regularization term is computed on basis of the first groiq) of pixels of the first two- 
dimensional block of pixels and a second group of pixels of the second two-dimensional 
block of pixels. 

7. A conversion unit (706) for converting a first set (100a) of initial segments of 
an image into a second set (100b) of updated segments (A',B',C',D') of the in:iage, the 
conversion unit being arranged to perform iterative updates of intermediate segments 
(A3,CJD) being derived from respective initial segments, a particular update comprising 
determining whether a particular pixel (300) being located at a border (302) between a first 
one of the intermediate segments (A), and a second one of the intmnediate segments (B), 
should be moved firom Ifae first one of the intermediate segments (A) to the second one of the 
intermediate segments (B), on basis of a pixel value of the particular pixel, on basis of a first 
parameter of the first one of the intermediate segments (A) and on basis of a second 
parameter of the second one of the intermediate segments (B), characterized in that the 
conversion unit (706) coir5)rises computation means for performing first a number of 
iterative updates for pixels of a first two-dimensional block of pixels (200) of the image and 
for, after that, performing the number of iterative updates for pixels of a second two- 
dimensional block of pixels (204) of the image. 

8. An image processing apparatus (600), comprising: 

- receiving means (602) for receiving a signal representing an image; 

- a segmentation unit (604) for determinmg a first set of initial segments of the 

image; 

- a conversion unit (606) for converting the first set of initial segments into a 
second set of updated segments, the conversion unit as claimed in claim 7; and 

- an image processing unit (608) for processing the image on basis of the 
second set of updated segments. 
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9. An image processing ^)paratus (600) as claimed in claims 

image iTOcessing unit (608) is designed to perforin video con^ 
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ABSTRACT: 



A method of converting of a first set (100a) of initial segments of an image 
into a second set of updated segments (A'3',C' J>') is disclosed. The method comprises 
iterative updates of intermediate segments (AJB^CiP) being derived from respective initial 
segments. Each update comprises determining whether a pixel (300) should be moved fix>m a 

S first intermediate segment (A) to a second intemiediate segment (B), on basis of a pixel value 
of the pixel, on basis of a first parameter of the intermediate segment (A) and on basis of a 
second parameter of the second intermediate segment (B). The iterative updates are 
performed on block base. That means that first a nmnber of iterative updates are performed 
for pixels of a first two-dimensional block of pixels (200) of the image and after that the 

10 number of iterative updates are performed for pixels of a second two-dimensional block of 
pixels (204) of the image. 



Fig. 2 
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